Goto

Collaborating Authors

 programming concept


Synthesizing High-Quality Programming Tasks with LLM-based Expert and Student Agents

Nguyen, Manh Hung, Pădurean, Victor-Alexandru, Gotovos, Alkis, Tschiatschek, Sebastian, Singla, Adish

arXiv.org Artificial Intelligence

Generative AI is transforming computing education by enabling the automatic generation of personalized content and feedback. We investigate its capabilities in providing high-quality programming tasks to students. Despite promising advancements in task generation, a quality gap remains between AI-generated and expert-created tasks. The AI-generated tasks may not align with target programming concepts, could be incomprehensible to students, or may contain critical issues such as incorrect tests. Existing works often require interventions from human teachers for validation. We address these challenges by introducing PyTaskSyn, a novel synthesis technique that first generates a programming task and then decides whether it meets certain quality criteria to be given to students. The key idea is to break this process into multiple stages performed by expert and student agents simulated using both strong and weaker generative models. Through extensive evaluation, we show that PyTaskSyn significantly improves task quality compared to baseline techniques and showcases the importance of each specialized agent type in our validation pipeline. Additionally, we conducted user studies using our publicly available web application and show that PyTaskSyn can deliver high-quality programming tasks comparable to expert-designed ones while reducing workload and costs, and being more engaging than programming tasks that are available in online resources.


Detecting Struggling Student Programmers using Proficiency Taxonomies

Schwartz, Noga, Fairstein, Roy, Segal, Avi, Gal, Kobi

arXiv.org Artificial Intelligence

Early detection of struggling student programmers is crucial for providing them with personalized support. While multiple AI-based approaches have been proposed for this problem, they do not explicitly reason about students' programming skills in the model. This study addresses this gap by developing in collaboration with educators a taxonomy of proficiencies that categorizes how students solve coding tasks and is embedded in the detection model. Our model, termed the Proficiency Taxonomy Model (PTM), simultaneously learns the student's coding skills based on their coding history and predicts whether they will struggle on a new task. We extensively evaluated the effectiveness of the PTM model on two separate datasets from introductory Java and Python courses for beginner programmers. Experimental results demonstrate that PTM outperforms state-of-the-art models in predicting struggling students. The paper showcases the potential of combining structured insights from teachers for early identification of those needing assistance in learning to code.


Web vs. LLMs: An Empirical Study of Learning Behaviors of CS2 Students

Kumar, Aayush, Prol, Daniel, Alipour, Amin, Ragavan, Sruti Srinivasa

arXiv.org Artificial Intelligence

LLMs such as ChatGPT have been widely adopted by students in higher education as tools for learning programming and related concepts. However, it remains unclear how effective students are and what strategies students use while learning with LLMs. Since the majority of students' experiences in online self-learning have come through using search engines such as Google, evaluating AI tools in this context can help us address these gaps. In this mixed methods research, we conducted an exploratory within-subjects study to understand how CS2 students learn programming concepts using both LLMs as well as traditional online methods such as educational websites and videos to examine how students approach learning within and across both scenarios. We discovered that students found it easier to learn a more difficult concept using traditional methods than using ChatGPT. We also found that students ask fewer follow-ups and use more keyword-based queries for search engines while their prompts to LLMs tend to explicitly ask for information.


Iris: An AI-Driven Virtual Tutor For Computer Science Education

Bassner, Patrick, Frankford, Eduard, Krusche, Stephan

arXiv.org Artificial Intelligence

Integrating AI-driven tools in higher education is an emerging area with transformative potential. This paper introduces Iris, a chat-based virtual tutor integrated into the interactive learning platform Artemis that offers personalized, context-aware assistance in large-scale educational settings. Iris supports computer science students by guiding them through programming exercises and is designed to act as a tutor in a didactically meaningful way. Its calibrated assistance avoids revealing complete solutions, offering subtle hints or counter-questions to foster independent problem-solving skills. For each question, it issues multiple prompts in a Chain-of-Thought to GPT-3.5-Turbo. The prompts include a tutor role description and examples of meaningful answers through few-shot learning. Iris employs contextual awareness by accessing the problem statement, student code, and automated feedback to provide tailored advice. An empirical evaluation shows that students perceive Iris as effective because it understands their questions, provides relevant support, and contributes to the learning process. While students consider Iris a valuable tool for programming exercises and homework, they also feel confident solving programming tasks in computer-based exams without Iris. The findings underscore students' appreciation for Iris' immediate and personalized support, though students predominantly view it as a complement to, rather than a replacement for, human tutors. Nevertheless, Iris creates a space for students to ask questions without being judged by others.


Do Large Code Models Understand Programming Concepts? A Black-box Approach

Hooda, Ashish, Christodorescu, Mihai, Allamanis, Miltos, Wilson, Aaron, Fawaz, Kassem, Jha, Somesh

arXiv.org Artificial Intelligence

Large Language Models' success on text generation has also made them better at code generation and coding tasks. While a lot of work has demonstrated their remarkable performance on tasks such as code completion and editing, it is still unclear as to why. We help bridge this gap by exploring to what degree auto-regressive models understand the logical constructs of the underlying programs. We propose Counterfactual Analysis for Programming Concept Predicates (CACP) as a counterfactual testing framework to evaluate whether Large Code Models understand programming concepts. With only black-box access to the model, we use CACP to evaluate ten popular Large Code Models for four different programming concepts. Our findings suggest that current models lack understanding of concepts such as data flow and control flow.


Boldly Going Where No Benchmark Has Gone Before: Exposing Bias and Shortcomings in Code Generation Evaluation

Yadav, Ankit, Singh, Mayank

arXiv.org Artificial Intelligence

Motivated by the increasing popularity of code generation from human descriptions using large language models (LLMs), several benchmarks have been proposed to assess the capabilities of existing and emerging models. This study presents a large-scale human evaluation of HumanEval and MBPP, two widely used benchmarks for Python code generation, focusing on their diversity and difficulty. Our findings reveal a significant bias towards a limited number of programming concepts, with negligible or no representation of most concepts. Additionally, we identify a concerningly high proportion of easy programming questions, potentially leading to an overestimation of model performance on code generation tasks.


C++ Developer openings in San Francisco Bay Area, United States on August 01, 2022

#artificialintelligence

Snap is proud to be an equal opportunity employer, and committed to providing employment opportunities regardless of race, religious creed, color, national origin, ancestry, physical disability, mental disability, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, pregnancy, childbirth and breastfeeding, age, sexual orientation, military or veteran status, or any other protected classification, in accordance with applicable federal, state, and local laws. If you have a disability or special need that requires accommodation, please don't be shy and contact us at_ _accommodations- • *About Us** Snap Inc. is a camera company. We believe that reinventing the camera represents our greatest opportunity to improve the way people live and communicate. Our products empower people to express themselves, live in the moment, learn about the world, and have fun together. At Snap, we believe that having a team of diverse backgrounds and voices working together will enable us to create innovative products that improve the way people live and communicate. Snap is proud to be an equal opportunity employer, and committed to providing employment opportunities regardless of race, religious creed, color, national origin, ancestry, physical disability, mental disability, medical condition, genetic information, marital status, sex, gender, gender identity, gender expression, pregnancy, childbirth and breastfeeding, age, sexual orientation, military or veteran status, or any other protected classification, in accordance with applicable federal, state, and local laws.


Python Programming - For Every Beginners

#artificialintelligence

This course is aimed at offering the fundamental concepts of core Python programming language to the all levels of learners. This course is aimed at offering the fundamental concepts of core Python programming language to the all levels of learners. You will learn through videos, visual organizers and practice exercises. For a great hands-on learning experience, this course is packed with assignments, assessment tests, code challenges, quizzes, and exercises. Part-1: Core Python Programming Basics- starts with the basics of Python programming concepts like introduction, history & versions, features, uses, applications of python, data types, operators and control flow statements.


100%OFF

#artificialintelligence

This course fully covers from classical C to Modern C style of creating object Oriented Programs from scratch to advance level in a step-by-step approach. The course teaches in detail the latest concepts introduced in C 11, C 14 and C 17. The object oriented programming concepts are covered in detail such that you will learn all the concepts including classes, objects, Data Abstraction, Data Encapsulation, Inheritance, polymorphism (including Operator overloading and Function Overloading). The main focus of the course apart from Fundamentals of programming and Object Oriented Programming is on Templates(including Function and Class Templates), which is a building block to understand STL implementation. The Course entirely covers all String Functions included in the latest version of C along with the basic programming concepts like operators, variables, Conditional statements and looping structures, functions(User-Defined and Recursive Functions), reference parameters, Arrays,File I/O and vectors in C .has been discussed in details.


The 7 Best Ways to Learn How to Code for Free

#artificialintelligence

You've probably come across the term'coding' plenty of times, and if you haven't, then this is the best place to start. As we progress into the 21st century, the need for code continues to increase. Coding used to be limited to computers and video games, but now it encompasses every part of our lives. Coding is now an essential part of most major industries such as healthcare, finance, engineering, etc. Read on as we walk you through the basics of coding and how you, too, can learn to code. Coding, in essence, is the ability to make a computer do a particular task through instructions written in a programming language.